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© A system and method to compensate for effects 
of a recording medium such as film on image data 
retrieved therefrom. Red, green, blue, and infrared 
light is sequentially transmitted through film having 
an image exposed therein to the system which cap- 
tures corresponding images. The infrared image 
maps the location, boundaries and intensities of the 
media defects. The map is employed combinatorially 
with the other captured images resulting in an image 
from the film free of the effects of the film defects. In 
one embodiment the effects of the imperfections are 
divided out from the red, green, blue spectra in an 
operation dividing out the infrared image data on a 
pixel-by-pixel basis. In an alternate embodiment 
wherein the imperfections substantially prevent 
transmission of IR light therethrough, the IR image 
map of the defects is nevertheless employed with an 
automated fill-in algorithm more effectively utilized 
due to the precision with which the defect bound- 
aries are known. 




FIG.1 
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Field of the Invention 



This invention relates to image enhancement 
and recovery, and, more particularly, to systems 
and methods which compensate for the effects of 
storage media defects on images stored therein. 

Background of the Invention 

Ever since the first image of an object was 
captured on film, a serious problem was apparent 
which has continued to plague the field of image 
capture and reproduction to the present day, name- 
ly imperfections in the recording medium itself 
which distort and obscure the original image 
sought to be captured. These imperfections occur' 
in innumerable forms including dust, scratches, fin- 
gerprints, smudges, and the like. Archival poly- 
propylene sleeves employed to protect negatives 
even contribute to the problem by leaving hairline 
surface scratches as the negatives are pulled out of 
and replaced into the sleeves. 

The problem is even worse with such negatives 
than with transparencies, first because viewing the 
negative requires a gamma, or ratio gain, of more 
than two, and secondly because filmstrips are more 
subject to contact than are mounted transpar- 
encies. Such imperfections may be found present 
even in fresh photographic film in surface waves, 
and may further arise from improper processing 
causing mild reticulation, and unbleached silver for 
example. Not only are such non-image imperfec- 
tions surface related but can even be caused for 
example by microbubbles in emulsion within the 
film itself. 

Numerous measures have been developed in 
the art in an attempt to address the problem, 
particularly with respect to captured high resolution 
images. One approach has been that of prevention 
as, for example, in the development process itself. 
Expensive anti-static equipment including spray 
machines to neutralize dust-attracting charges are 
employed for example by reputable photo finishers. 
Photo finishers also attempted to employ diffuse 
light source enlargers that helped reduce the ef- 
fects of refraction of the light by imperfections in 
the photo finishing processes. 

Yet another way the problem was approached 
sought to minimize the effects of these imperfec- 
tions once they were present by various correction 
techniques, most of which were manual and thus 
highly labor-intensive and expensive. As an exam- 
ple, during the photo finishing process, a highly 
trained individual might spend a great deal of time 
with various spotting dyes and an extremely small 
spotting brush seeking to essentially paint out the 
imperfections. Another technique was to wipe on or 
immerse the negatives in a light oil in an attempt to 



optically fill scratches. 

The problem of dust, scratches, and the like in 
large measure contributed to the decline in use and 
popularity of vinyl records. The film industry has 

5 been concerned that the problem may in like man- 
ner jeopardize the long term future of analog im- 
ages. Notwithstanding the significant efforts that 
were made to solve the problem, it seemed that no 
matter what was done the problem nevertheless 

70 persists. This is particularly with respect to enlarge- 
ments and high resolution scans. Thus the problem 
is becoming even more acute and recognized as 
resolution increases and multimedia brings atten- 
tion to it with the increase in film scanning for 

is computer applications. 

As the art developed, various attempts were 
made to automate the correction process, particu- 
larly with respect to digital image systems.. In such 
systems once an imperfection was detected, var- 

20 ious "fill" algorithms had been developed for cor- 
recting the image at the situs of the imperfection. 
Nevertheless, heuristics or human intervention were 
required to detect the imperfections with a subjec- 
tive threshold. Typically the identified area to be 

25 corrected in this manner was much larger than 
necessary under the present invention, in part due 
to these subjective criteria for detecting defective 
areas. 

Automated methods were in fact developed for 

30 even detecting imperfect areas in recording media, 
notably as described in the German patent 
#2821868.0, published 11/22/79, entitled "Method 
and Device for Detecting Recording and Counting 
of Mechanical Damage to Moving Bands, for Exam- 

35 pie, Films". In this system, a source of infrared 
energy impinged upon the film medium. A scanned 
infrared image thereafter was taken of the film in 
question by sensors detecting reflection of the in- 
frared energy from the film surface. However sev- 

40 eral limitations were present in this system. 

First, its purpose was not to correct an image 
present on the film for the effects of such detected 
film defects. Rather the system was implemented 
simply to monitor the prevalence of these defects 

45 in an automated photographic development pro- 
cess whereby, for example, the process could be 
automatically shut down if the defect rate exceeded 
a prescribed level. Moreover, infrared image was 
not recorded in registry with any other images from 

so the visual portion of the electromagnetic spectrum. 
This alone would render it extremely difficult to 
subtract out the effect of such imperfections noted 
in the IR image from similar defects present and 
visible in the visual record of the image. On the 

55 contrary, the optical IR path was a reflective one 
from the IR source to sensor different from the 
other sensors unutilized for image processing. 
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Such systems have in fact been developed for 
capturing images of objects along an identical op- 
tical path wherein each such image corresponds to 
a different portion of the electromagnetic spectrum, 
notably in the satellite photography and reconnais- 
sance arts, for example, for purposes of image 
enhancement. In such systems, images of one 
spectrum may thus be superimposed on those 
from another spectrum of the identical object 
whereupon various image processing algorithms 
well known in the art might thereafter be employed. 
For example, an infrared mapping system might 
functionally relate the IR spectrum of carbon diox- 
ide in an earth surface image from that of another 
portion of the spectrum to draw conclusions re- 
garding surface vegetation or the like as, for exam- 
ple, in crop inventorying; and such techniques have 
been well developed. However, these systems 
while in some sense enhancing the captured im- 
age, are not detecting through means of the in- 
frared spectrum defects and imperfections asso- 
ciated with the recording medium itself, and there- 
after using such information to enhance the image 
stored on the medium. 

A object of the invention is to render invisible 
or substantially reduce the effect of physical imper- 
fections of a storage medium caused by dust, 
scratches, and the like, from the desired image 
stored on the medium. 

The present invention may be better under- 
stood with reference to the following figures. 

Brief Description of the Drawings 

Rg. 1 is an illustration of the separation and 
recombination of images in different spectra of 
an object on film having imperfections in order 
to reduce their effect in the resultant combined 
image; 

Fig. 2 is a schematic representation of a repre- 
sentative system for implementing the invention; 
Rg. 3 is a functional block diagram of portions 
of the system of Fig. 2; 

Rg. 4 is a flow diagram indicating a sequence of 
operation of the system of Fig. 2 in accordance 
with the invention; 

Rg. 5 is an illustration of various frequency 
response signatures associated with aspects of 
the invention including that of recording media 
imperfections, a typical array for capturing im- 
ages from the medium, and that of various dyes 
associated with the dye-based film-type form of 
the recording medium; 

Rg. 6 is a schematic illustration of image pro- 
cessing steps carried out by the system of Fig. 
2 which comprise a portion of the flow diagram 
of Fig. 4; 



Fig, 7 is a schematic illustration of a piece of 
film containing a defect obscuring an underlying 
image; 

Fig. 8 is another illustration of the film of Fig. 7 
5 depicting the results of two prior art methods for 
correcting images having defects caused by re- 
cording medium imperfections; 
Fig. 9 is another illustration of the film of Fig. 7 
depicting results of the invention in correcting 
w the image in Fig. 7 for the defects shown there- 
in. 

Summary of the Invention 



75 A system and method for use in compensating 

for effects of storage media defects on image data 
retrieved therefrom. Image data is stored on a 
recording medium containing non-image imperfec- 
tions such as film having surface scratches, waves, 

20 smudges, bubbles, or the like, which give rise to 
undesirable blemishes in images subsequently re- 
trieved from the medium. 

Means are provided for deriving from the me- 
dium separate images in the red, green, blue, and 

25 infrared portions of the electromagnetic spectrum 
corresponding to the image stored therein. 

In one embodiment, red, green, blue, and in- 
frared light is sequentially directed at one surface 
of the film by means of a light source and color 

30 filter wheel. Corresponding sequential red, green, 
blue, and infrared images formed by that portion of 
the light being transmitted through the film are 
digitally captured from the opposite side of the film. 
The images are preferably captured in registry to 

35 facilitate subtracting out effects of imperfections at 
locations in the infrared record from corresponding 
locations in the red, green, blue images. The im- 
perfections may either substantially reduce or to- 
tally occlude the infrared light. However, remaining 

40 portions of the medium having the desired image 
without such imperfections are essentially uniformly 
transmissive to the infrared spectrum while of vari- 
able transmissivity in the visual spectrum as deter- 
mined by the image developed on the film. Accord- 

45 ingly, the infrared image may serve as an indicator 
or map of the spatial position of these non-image 
imperfections on and in the media, thereby allow- 
ing recovery of the underlying desired image. 

In the case wherein total occlusion does not 

50 occur, inasmuch as imperfections are essentially 
colorless, the amount of reduction in infrared trans- 
mitted through the medium at known locations on 
the film corresponding to the imperfections can be 
utilized to increase intensity of the red, green, blue 

55 images at these same locations so as to cancel the 
effect of the imperfections. Alternately, in the case 
of total occlusion, the infrared record provides an 
accurate map of the location and shape of the 
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imperfection to which conventional fill-in algorithms 
are applied. 

Detailed Description of the Preferred Embodiment 

Referring to Fig. 1, first a generalized descrip- 
tion of phenomenon associated with the invention 
will be provided in order to facilitate its understand- 
ing. In the photographic film art, typical dyes in film 
or transparencies are transparent to the infrared 
spectrum, whereas imperfections in the film are 
not. Thus an infrared image derived from such a 
film with these physical non-image imperfections 
will only contain images of these non-dye imper- 
fections. A map of such imperfections from this 
infrared image may thence be employed to math- 
ematically cancel their effects, and thus derive the 
desired pure dye image taken from the film. 

As hereinbefore set forth, numerous imperfec- 
tions are associated with film, such as dust, 
smudges, and the like. The terms film and trans- 
parency are utilized interchangeably and broadly 
herein and are intended but not to be limited to 
include any film media for storing images which 
passes light including negatives as well as posi- 
tives. Such imperfections characteristically are al- 
most always colorless and thus affect the infrared 
spectrum equally with the red, green, and blue 
components of the visible images retained in the 
film. This is true of various forms of such imperfec- 
tions. For example, small surface scratches on film 
are visible because they refract light out of the 
visual path. This refraction, although very slightly 
greater for light of short wavelengths, nevertheless 
is effectively independent of wavelength. Finger- 
prints and smudges on film are deposits of trans- 
parent oils, and again refraction alone makes them 
visible. In like manner, included bubbles in the 
emulsion or mild reticulation due to improper film 
processing again are visible because of refraction, 
having essentially no color themselves. Dust is yet 
another foreign agency frequently contributing to 
degradation in the quality of images retrieved from 
film media. Most dust is, in like manner, substan- 
tially transparent when viewed as a single filament, 
having a color only when viewed as the sum of 
many fibers, and again is visible primarily through 
refraction. In like manner, metallic flecks are opaq- 
ue, and opaque particles will affect all colors sub- 
stantially equally, subtracting out an equal percent- 
age of light from the pel on which they lie propor- 
tionate to their size, but almost independent of 
wavelength. It is rare from the foregoing that one 
may find dust or other particulate matter on film 
which is substantially colored as an individual fila- 
ment requiring additional processing to remove the 
effects thereof from the image beyond those con- 
templated by the invention. 



Because of this almost universal colorlessness 
of film imperfections, it will appreciated that a 
"dust" image in the infrared spectrum may be 
derived from film containing undesired defects and 
5 imperfections, and this same image may be ap- 
plied to red, green, and blue records also obtained 
from the same portion of film. These imperfections 
may therefore be seen in the infrared, and this 
single dust image applied to the three colored red, 
70 green and blue records. As a refinement, the dust 
image may be applied as hereinafter set forth in 
greater detail with slightly more strength in the blue 
record and slightly less strength to the red record 
in order to compensate for the slight color sensitiv- 
75 ity of refraction and defraction. 

Most imperfections associated with film media 
reduce light transmitted therethrough to some ex- 
tent. In accordance with the teachings of the inven- 
tion, by dividing out the amount of reduction across 
20 the red, green and blue images, the effects of the 
imperfection are thereby nulled on the image de- 
sired to be retrieved from the film. As will also be 
hereinafter detailed, this utilization of the infrared 
record on the red. green, and blue spectra will not 
25 be entirely uniform across each spectrum due to 
the aforementioned non-linearities associated with 
refraction and defraction, different sensitivities of 
film dyes and light sensors to differing spectra, and 
the like. Such details will be described further, but 
30 for present purposes, in order to understand the 
invention generally, it may be conceptually under- 
stood that the infrared record will essentially be 
utilized to adjust the affect of the imperfections 
uniformly in the red, green, and blue spectra. 
35 Although most imperfections associated with 

film media will reduce light to some extent as 
aforesaid, in some instances the imperfection may 
totally occlude a part of the desired image. In such 
instances in accordance with the invention, when 
40 the effects of the imperfection may not be 
"retracted" by employing the IR map with the red, 
green, and blue spectra, then the invention at least 
provides an extremely accurate detection of the 
exact shape and location of the various imperfec- 
45 tions in the film, thereby permitting other fill-in 
algorithms well known in the art to be employed. A 
significant limitation of those techniques was the 
inability of such algorithms to accurately distinguish 
imperfections from image detail. Accordingly, im- 
' so age detail was often lost due to the heuristic, 
subjective identification of the situs of imperfec- 
tions which was frequently imprecise due to the 
human intervention. In accordance with the inven- 
tion as will hereinafter be described, in these cases 
55 where imperfections substantially occlude the im- 
age, the system and method described herein may 
be used in conjunction with such fill-in algorithms 
to automatically render the imperfection effectively 
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invisible because their locations and boundaries 
are now more precisely identified without unneces- 
sarily losing image detail. 

Fig. 1 illustrates the foregoing general princi- 
ples of the invention just discussed. A conventional 
film transparency 12 has one or more frames 10 
which include both an image 14 and physical im- 
perfections or defects 16 previously described as- 
sociated with the film 12, such as dust, scratches, 
or the like, which impair the ability to retrieve the 
pure image 14 from the film 12. In accordance with 
the invention, a mechanism such as a color wheel 
or the like (shown conceptually as a prism 18) will 
be employed to separate the information compris- 
ing the image 14 and imperfections 16 into a 
plurality of individual records, each associated with 
a different portion. of the electromagnetic spectrum. 
Specifically, in sequence infrared 20, red 22, green 
24, and blue 26 images of the film 12 are captured. 

Referring first to the infrared image or record 
20, a comparison of the pattern of imperfections 16 
appearing in the film 12 with those appearing in the 
infrared record 20 will reveal that they are substan- 
tially identical in spatial positioning. As previously 
indicated, these imperfections 16 associated with 
film 12 will impede transmissivity of the infrared 
through the film 12, and thus these imperfections 
appear at locations 16 on the infrared record 20. It 
will also be noted in the infrared image 20 that the 
components comprising the image 14 resulting 
from various photographic dyes exposed in the film 
12 do not appear in the image 20. The reason for 
this is that, again as previously noted, these dyes 
are essentially transparent to infrared light transmit- 
ted through the film 12. 

Referring now to the remaining red, green, and 
blue images 22-26, respectively, each such image 
will be seen to pick up and record portions of the 
image 14 of the frame 10 associated with their 
respective visible spectra. For example, in the 
green record 24 a number of leaves may be seen 
which, in a color illustration would have generally 
green hues. However the remaining non-green por- 
tions of the image 14 are expectedly absent in the 
green record 24. This is an example of conven- 
tional color separations well known in the art. How- 
ever, again comparing the location of imperfections 
16 appearing in the red, green, and blue images 
22-26 will reveal that they again are spatially posi- 
tioned in the image essentially in identical locations 
to those in which they appear in the original film 
12. Moreover these imperfections 16 appear in 
each of the red, green, and blue records, again 
bearing out the previously described property" of 
such imperfections 16, namely that they are essen- 
tially colorless and thus impede transmission of 
red, green, and blue light through the film 12 
substantially equally. 



The operation of the invention generally may 
now be clearly seen by comparison of these four 
records 20-26. Because none of the desired visual 
image 14 appears in the infrared record 20, al- 
5 though the imperfections 16 appear in all four 
records (and particularly in the records of the visi- 
ble spectrum otherwise impairing the ability to re- 
construct the image 14 without such imperfections 
16), a subtractive process may be employed to 

io reconstruct the image 14 from images 20-26 as a 
new image 36 wherein the imperfections 16 are 
substantially reduced or eliminated. Specifically, as 
shown conceptually by the additive processes 28, 
30, and 32, and subtractive process 34, each red, 

75 green, and blue image is added together resulting 
in the improved image 36. However, the infrared 
image or "map" of the undesirable imperfections 
16 are utilized with respect to each of these visual 
images 22-26 to effectively mathematically subtract 

20 out (as shown by process 34) the undesirable 
affect of these imperfections 16 on each of the 
records in the visual spectra. 

For simplicity's sake at present, one may ap- 
preciate that if the imperfections 16 show up as 

25 imperfections of equal intensity in the red, green, 
and blue images (due to the uniform transmissivity 
of the imperfections 16), given that the precise 
location of these physical imperfections 16 is 
known from the infrared record 20 (which includes 

30 no information from the desired image 14 to be 
reproduced), these locations of the imperfections 
16 in the infrared record 20 may be utilized to 
uniformly increase the exposure or intensity of pix- 
els at these same precise locations in the red, 

35 green, and blue records in a reciprocal way to the 
infrared intensity to result in the improved image 
36. Thus, the red, green, and blue images are 
recombined (as shown by summing functions 28- 
32) to result in the image 36, but each such record 

40 first has subtracted out (as shown by the subtrac- 
ting function 34) the intensities at the pixel loca- 
tions of the imperfections 16 appearing in the in- 
frared record 20 and the corresponding red, green, 
or blue image 22-26. 

45 Turning now to Fig. 2, a schematic illustration 

is provided of a representative system for both 
capturing the necessary four images 20-26 of Fig. 
1 from the frame 10 of film 12, and thereafter 
recombining them as desired in the manner just 

so described to yield an improved image 36. The 
conceptual effect of the prism 18 will in the em- 
bodiment depicted in Fig. 2 be implemented by 
means of a color wheel 38 having a plurality of 
optical filters 40, specifically a red, green, blue and 

55 infrared filter. A light source such as a lamp 42 
provides light in each of these red, green, blue and 
infrared spectra. With each of the appropriate color 
filters 40 being sequentially interposed between the 
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film 12 and the lamp 42, light in the red, green, 
blue and infrared spectra will sequentially be caus- 
ed to impinge upon and be transmitted through the 
desired frame 10 of the film 12 in sequence and 
will thence be captured by an appropriate color 
camera 42, whereupon the video signal 46 thus 
generated may be delivered to a suitable computer 
48. 

The process of generating sequential color im- 
ages may of course be automated as shown gen- 
erally by the control line 44 from the computer 48 
to the image generating component of the system 
of Fig. 2. The functions which may be controlled by 
the computer 48 and control line 44 are well known 
in the art and will thus not be discussed in detail 
herein. Such functions may include control signals 
to peripheral devices to sequence advancement of 
the respective color filters 40 on the color wheel 
38, advancement of the film 12 to the desired 
image 10, and may further include other such 
controls features as controlling intensity of the lamp 
42 for the desired spectral response. A keyboard 
52 provides a conventional user interface to the 
computer and a color monitor 50 is further pro- 
vided for viewing the various images as desired. 

Referring now to Fig. 3. the computer 48 will in 
a conventional manner include a system bus 54 for 
providing intercommunication between a micropro- 
cessor 56 within the computer and a program 58 
and input/output adapter 60. The adapter 60 which 
mav take the form of a video capture card is 
pre ded in order to interface with the camera 42 
anc monitor 50 so as to provide proper digitized 
image data to be stored in and operated upon by 
the computer 48. Additional adapters may be pro- 
vided as desired in order to provide further system 
control and interfacing as for example in provision 
of " a control signals 44 dependent upon the par- 
tic^:;! system employed. The computer 48 will 
des-: - aDly include some form of bulk memory ca- 
pable of storing large amounts of data associated 
with high resolution digital images, such storage 
being conceptually illustrated in Fig. 3 as the im- 
age blocks 62-66. Blocks 62 and 64 are intended 
to schematically represent the various red, green, 
blue and infrared images captured from the frame 
10 and stored in the form of digitized pixels in 
memory. In like manner the output image box 66 is 
intended to functionally represent the recombined 
visible image 36 as stored in memory after the 
computer 48 has operated upon the input images 
62 and 64 in accordance with the invention. This 
output image 64 may also preferably be in the form 
of a plurality of digitized image pixels stored in 
bulk memory in a manner well known in the art. 

Still referring to Fig. 3, the program 58 is 
intended to represent software necessary for con- 
trolling the microprocessor 56 and other compo- 



nents of the system in the desired manner to 
achieve the objectives described herein. For exam- 
ple, a component of the program 58 may be the 
control software necessary for generating sequen- 

5 tially the desired red, green, blue and infrared 
images. Microprocessor 56 under the control of this 
program 58 will thus generate the required signals 
on the system bus 54 transmitted to the 
input/output adapter or adapters 60 for placing ap- 

70 propriate signals or receiving such signals on the 
lines 44 and 46 which may include 
data'address/control signals as necessary. Yet an 
additional component of the program 58 will en- 
code the steps necessary to carry out the process- 

75 ing of the images as herein described. 

One representative computerized system for 
implementing the functions just described relative 
to Figs. 2 and 3 would include the following com- 
ponents: 

20 The lamp, filter wheel, film holder, and camera 

assembly is contained within a Nikon film scanner 
model LS 3500. The scanner is modified so during 
the infrared scan the built-in infrared filter is re- 
moved, a Kodak Wratten 87C filter inserted in its 
25 place, and the voltage on the lamp lowered from 
the normal 12 volts to 5 volts. 

The computer, display and keyboard can be 
one manufactured by International Business Ma- 
chines containing a GPIB card manufactured by 
30 National Instruments of Austin, Texas to interface 
to the film scanner. 

Alternately the camera 42 can be a Puinix 
monochrome CCD camera model TM34K specially 
ordered without the built-in infrared absorbing filter, 
35 the colored filters 40 include an 87C for infrared 
and standard colors for the other positions plus 
added infrared absorbing filters. The arrangement 
of the wheel, lamp, and film holder should be 
obvious to one skilled in the art. 
40 With reference to Fig. 4, the pertinent portion 

of the program 58 which accounts for the imperfec- 
tions 16 and the infrared image 20 in processing 
the various images to result in the desired recom- 
bined image 36 will now be described. Once the 
45 red, green, blue and infrared images have been 
digitized and appropriately stored for retrieval, the 
image processing will commence by retrieving 
pixel data as shown by arrow 68. Each pixel digitis- 
ed and captured from the frame 10 will actually 
50 include a corresponding red, green, blue and in- 
frared pixel associated with the respective red, 
green, blue and infrared images 20-26 of Fig. 1. 
These red, green, blue and infrared pixels cor- 
responding to the same location in the frame 10 
55 will have intensity values associated therewith 
which under computer control will be stored and 
retrieved from memory. For each such particular 
group of pixels the associated value of the cor- 
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responding infrared pixel 76 will be compared to a 
predetermined threshold value as shown by de- 
cision tree 78. This threshold valve is selected 
based on the noise and accuracy of the actual 
hardware to limit subsequent compensating gain 
below the point where system noise predominates. 
This limit is conceptually equivalent to the limit in 
the automatic gain control (AGC) of a radio re- 
ceiver. 

If the value of this infrared pixel does not equal 
the predetermined value, the pixels associated with 
this infrared pixel and corresponding location in the 
frame 10 are considered to be obscured by an 
imperfection 16 and this fact is recorded with re- 
spect to the pixels as shown at block 84. An 
appropriate fill routine may then be executed by 
the system of Fig. 2 to replace the respective red, 
green, blue and infrared pixel values with corrected 
estimates therefor more accurately reflecting what 
the values would be if the corresponding imperfec- 
tions 16 were absent from the film 12. Such fill-in 
routines are well known in the art and employ 
various techniques such as examining values of 
adjacent pixels in order to extrapolate to an es- 
timated value for the obscured pixel as shown at 
step 86. 

Continuing with Rg. 4, if the pixel value of the 
particular infrared pixel under examination is equal 
to or exceeds that of the predetermined value, the 
branch 82 of the decision tree 78 is followed. As 
previously described, under control of program 58, 
the computer 48 will thereafter adjust the cor- 
responding red, green and blue pixel values re- 
trieved in blocks 70-74 by values functionally re- 
lated to that of the infrared pixel intensity as shown 
at blocks 90-94. It will be recalled that because the 
imperfections 16 will essentially, due to their uni- 
form transmissivity across the visual spectrum, 
manifest themselves as dimmed areas perceived in 
the red, green and blue records, these intensities 
must be appropriately intensified so that the areas 
associated with the imperfections will be in. har- 
mony with those of immediately adjacent pixels, 
thereby reducing the perception of these imperfec- 
tions in the enhanced image 36. Details of the 
precise mathematical operation in which the red, 
green and blue values are "divided" by the infrared 
value of the corresponding pixel will be hereinafter 
described in greater detail. 

Once each red, green and blue pixel has been 
compensated for the effect of the imperfections 16 
by division in accordance with blocks 90-94, or 
marked as observed by block 84, the estimation or 
fill-in techniques (block 86) may be employed for 
the pixels marked observed. The subroutine is then 
exited at 88. When the foregoing process has been 
completed for all captured pixels of a frame 10, a 
resulting output visible image record 66, Fig. 3, is 



thereby generated comprised of red, green and 
blue digitized images each comprised of its re- 
spective collection of pixels thus adjusted as re- 
quired in accordance with the process of Fig. 4. 
5 These corrected red, green and blue images may 
thereafter be selectively recovered on the system 
bus 54 from memory under control of microproces- 
sor 56 and thereafter output through the I/O adapt- 
er 60 in appropriate form whereby they may be 

70 combined as, for example, in the monitor 50 so as 
to display the desired recombined and enhanced 
image 36 devoid of the deleterious visual affect of 
the imperfections 16. 

Now that a general understanding has been 

75 provided of the operation of the invention, a more 
detailed description of important aspects thereof 
follows. Numerous factors must be accounted for in 
a practical image enhancement system relating to 
limitations and non-linearities of the real world 

20 which result in compromises over ideal theoretical 
systems. Fig. 5 shows the relative absorption of 
dyes and imperfections such as dust and 
scratches, and relative sensitivity of image capture 
sensors and transducers such as charge coupled 

25 device (CCD) arrays to various frequencies of light, 
where the ordinant is the measure of relative per- 
centage of absorption, or for the sensor, sensitivity, 
and the abscissa is the light wavelength in nano- 
meters. Dye colors are labelled with two names, for 

30 example "red/cyan" dye is formed by the. red 
sensitive layer to absorb red light, therefore ap- 
pears cyan in color. 

Several things may be noted from Fig. 5. First, 
as previously described, imperfections 16 such as 

35 dust and the like exhibit substantially uniform ab- 
sorption of light across the visible range between 
approximately 400-700 nanometers and into the 
infrared spectrum from about 700 nanometers on 
up. Accordingly, the graphical depiction of absorp- 

40 tion of these imperfections is substantially flat as 
shown at reference numeral 92. Because the pri- 
mary effect from most imperfections is refraction, 
and refraction is slightly less at longer wavelengths, 
the effect of most imperfections will be slightly less 

45 at longer wavelengths, shown in Fig. 5 as a slight 
tilt to absorption curve 92. Moreover, as mentioned 
in the background of the invention, a more diffuse 
light, referred to in microscopy as a light with a 
larger "numerical aperture", or NA, will also lower 

50 the effect of refraction-induced defects. Thus by 
constructing an illumination source that is less dif- 
fuse, or of a lower NA, in the red and especially 
infrared, and slightly more diffuse in the green, and 
especially blue, the tilt of absorption curve 92 may 

55 be removed. 

The invention has particular application in dye- 
based films. However, unique properties associated 
with such dyes create problems which must be 



7 



13 



EP 0 569 142 A1 



14 



accounted for. For example, referring to Fig. 5, one 
characteristic of these dyes is that some cyan dyes 
absorb some infrared. Cyan dyes used in positive 
transparencies are designed to work with the hu- 
man visual system, and accordingly are permitted 
to open up rapidly even in the near infrared, as 
shown by the positive transparency red/cyan dye 
response curve 98. On the other hand, cyan dyes 
utilized in negatives typically work with colored 
paper having a cyan-forming layer sensitive almost 
into near infrared, and accordingly one must go 
further into the infrared for these negative cyan 
dyes to become transparent. This is demonstrated 
by the spectral response curve of red/cyan dyes 
utilized for negatives, reference numeral 100. 

Also shown in Fig. 5 is a light frequency re- 
sponse curve 90 of a charged couple device array 
which is a typical transducer for capturing light 
images both in the visible and infrared spectra. 
Such a device might typically be found in the 
camera 42 of Fig. 2 for capturing the red, green, 
blue and infrared images. It will be plainly seen 
from the spectrum 90 there is no problem electron- 
ically in "seeing" infrared as the sensitivity of such 
arrays extend into the infrared spectrum and in fact 
frequently require eliminating the invisible infrared 
from a color image. As shown in the response 90, 
such CCD sensors are often more sensitive to 
infrared than visible light, and thus may adequately 
be employed to derive the desired infrared image 
20 of Fig. 1 when the frame 10 is illuminated with 
an incandescent form of lamp 42 which puts out 
most of its radiance in the infrared. 

A problem in implementing the system of Fig. 
2 might arise if a cold light source is employed 
exclusively as the lamp 42 or if an infrared absorb- 
ing filter is mechanically built into the light sensor 
of the camera 42, as it would in, for example, in a 
single chip color CCD array. Optically, such CCD 
sensors permit more light piping in the infrared, 
softening the image and possibly requiring a soft- 
ware boost of high spatial frequencies to renor- 
malize it to the visible acutance. Lenses associated 
with the camera 42 also focus differently in the 
infrared, although this problem may be minimized 
with apochromats or may be eliminated by simply 
refocusing slightly, which may even be done 
through computer control on the I/O control line 44 
automatically. Lateral color distortions are also po- 
tential problems with asymmetric lens designs 
which may be solved by software micro resizing or 
by using a symmetric lens design. It has been 
found in practice that such lens focus problems are 
small enough to ignore within 100 nanometers of 
the visible spectrum. Also referring to Fig. 5, it will 
be noted that conventional blue/yellow dye spectra 
such as the representative spectrum 94 and 
green/magenta spectra such as that shown at refer- 



ence numeral 96 do not absorb in the infrared 
region. 

It will be recalled that it is a principle of the 
invention that the infrared record will include data 
5 only associated with the location of the undesirable 
imperfections 16. These imperfections 16, because 
of the essentially flat transmissivity curve 92 there- 
of extending into the visible region will cause per- 
ceptible imperfections to appear in the respective 
io blue, green and red images as darkened areas. 
However, because an infrared record exists map- 
ping the locations and darkness of these imperfec- 
tions, this information in the infrared map may be 
utilized to compensate out the imperfections ap- 
75 pearing in the blue, green and red images. One 
purpose in providing Fig. 5 is to illustrate that in a 
non-linear world, to compensate these imperfec- 
tions, a practical system of the invention must 
account for these non-linearities and irregularities in 
20 dye and senor response, optics, etc. Thus, choos- 
ing the wavelength to capture the infrared record is 
in itself a compromise as may be understood from 
the foregoing. If cut too close to the visible spec- 
trum, excessive crossover from the cyan dye 
25 record will cause problems in the separation. On 
the other hand if cut too far into the infrared, the 
optical problems previously mentioned are made 
worse. Accordingly, it has been found in practice 
that a low pass filter below 750 nanometers is a 
30 good compromise. The blue sensor will see some 
of the magenta and cyan record as well as the 
yellow record it is supposed to see. Similarly, the 
infrared sensor will see some of the cyan record as 
well as the "dust" record it is supposed to see. 
35 From the four sensor variables it is of course 
desired to find the "pure" original red, green and 
blue records having information associated with the 
"pure" image, and further to derive a dust record 
devoid of all image information. In algebraic ter- 
40 minology, this suggests four variables and four 
unknowns. Although there are non-linearities in im- 
plementing the system of the invention as just 
described, in one embodiment acceptable separa- 
tion may actually be accomplished even by apply- 
45 ing a simpler linear algebra routine utilizing a pre- 
determined 4x4 matrix as shown in Fig. 6. 

Referring in more detail to Fig. 6, the purpose 
of this figure is to illustrate, given the foregoing 
discussion of non-linearities and the real world, that 
50 the image processing, once the red, green and 
infrared records have been derived, may take a 
number of forms in co-relating the infrared to the 
respective red, green and blue records in a manner 
determined by the particular system being em- 
55 ployed and the results desired. Specifically, the 
invention admits to a linear algebra matrix ap- 
proach in its simplest form but contemplates other 
processing routines as well. 
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Referring in more detail to Fig. 6, the values of 
individual pixels for each pixel location associated 
with the frame 10 will preferably be operated upon 
by the system of Fig. 2 and specifically the micro- 
processor 56. Each value of red, green, blue and s 
infrared 102-108 associated with a particular pixel 
will in one implementation have the cube root 
thereof obtained as shown in block 110. whereupon 
the results will be operated upon by a linear trans- 
form 112. The reason for the linear transform 112 w 
is to simplify the matrix algebra so the calculations 
may be done faster on a computer. The coeffi- 
cients 113, will, in one embodiment, be selected 
using method of linear algebra to provide indepen- 
dence of all four colors near a selected reference, 75 
such as gray. The resulting outputs of the trans- 
form 112 will then be cubed as shown by boxes 
114 resulting in red, green, blue and infrared val- 
ues 116-122, respectively, Once these cubed val- 
ues 116-122 have been obtained, to transform from 20 
the cube root domain in which the matrix was 
applied, the result for the imperfection or dust 
record 122 for the particular pixel in question will 
be employed as shown by the divide functional 
boxes 124 to reduce or divide out the effect of the 25 
imperfections in the red, green and blue records. 
The gain of the dust record 122 may be adjusted 
slightly to match the refraction of each color with 
gamma gain boxes 123. These values are then 
directly output as R, G, B values 130-134 in the 30 
absence of interpolation 126. 

As shown in Fig. 4, if the value for a given pixel 
in the infrared record, 122, is tested as shown in 
the threshold box 128 and is found to be below the 
threshold of acceptable recovery, this signifies that 35 
the particular pixel is obscured, thereby activating 
an appropriate interpolation or fill-in routine 126 
instead of the divide functions 124 to adjust the 
values for the red, green and blue pixels in accor- 
dance with values for adjacent pixels, thereby re- 40 
suiting in the correct red, green and blue pixel 
values 130-134. The interpolate box 126 receives 
outputs from divide 124 so that fringe areas of the 
imperfection that are partially, but not totally oc- 
cluded may be used in the estimate. This will be 45 
covered later in more detail referring to figs. 7 and 
9. 

Separation with a linear matrix 113 is an expe- 
dient greatly simplifying computation, however 
such an assumption of linearity introduces errors in 50 
areas of color, these errors can be reduced to the 
point of insignificance by applying matrix multipli- 
cation in a domain of selected non-linearity. To 
select this linearity, several cases may be consid- 
ered. First, in cases in which it is assumed the 55 
dyes may have a perfect square spectral response, 
e.g. they completely transmit some wavelengths 
and absorb a fraction of light at other wavelengths 



independent of wavelength. In this case it may be 
further assumed that each color sensor in the cam- 
era 42 may have a response covering some broad 
range of wavelengths, as they typically do in the 
real world. In such a case just described, a linear 
form of the matrix 112 separates out effects of the 
imperfections perfectly when applied in the linear 
domain. 

However, in yet another case in modelling to 
take into account non-linearities, it may in some 
instances be assumed that absorption of the dyes 
varies with wavelengths, as in fact is more typical. 
In some cases it may further be assumed that the 
color sensors are sensitive only to one wavelength 
as they would be, for example, if the scanning 
system of Fig. 2 employed four colored lasers as 
light sources. In this latter case, the linear matrix 
112 separates perfectly when applied in the loga- 
rithmic domain. 

Yet an additional case may be considered 
wherein dye sensitivity varies with wavelength and 
sensors cover some broad range of wavelengths. 
The linear matrix 112 may not separate perfectly, 
but nevertheless may function sufficiently to 
achieve effectively perfect separation of the effects 
of the imperfections 16 if applied in a domain 
between the linear and logarithmic. For this pre- 
ferred embodiment, the cube root domain as 
shown in Fig. 6 has been employed and appears to 
work quite well. 

The infrared or "dust" record absorbs 
wavelengths so uniformly as to fall into the first 
case most typically, and thus is best separated 
from the color records by a linear subtraction in the 
logarithmic domain. This is precisely a division in 
the linear domain and as such is shown by division 
blocks 124 in Fig. 6. Thus, the dye records are 
subtracted from the dust record in the cube root 
domain, and then the dust record is divided from 
the dye records in the linear domain. If the tilt of 
the dust/scratch absorption 92 is not compensated 
with variable diffusion as mentioned earlier in the 
specification, then the effect of imperfections will 
be slightly greater in the blue image than in the red 
image. To compensate this, the dust pixel valve 
122 can be raised to a power "gamma" prior to 
dividing the red, green, and blue image. Typically 
the best "gamma" for red is 1.03, for green, 1.06, 
and for blue, 1.1 

However, when the dust record falls below a 
predetermined threshold as indicated at block 128, 
interpolation may be used in place of division. It 
has been found that a pyramid interpolation will in 
practice remove small imperfections to the point of 
invisibility. Large imperfections on the other hand 
may be masked by an algorithm which duplicates 
texture as well as density across the defect, such 
as a fractal generator. However, employing such a 
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technique may be desirable only in the case of 
extremely large defects. 

Turning now to Figs. 7-9, these figures are 
intended to illustrate comparatively the results of 
image enhancement employing prior art methods 
with those of the present invention. In these figures, 
an identical desired image 136 is stored. A piece of 
film 139 contains a defect 138 obscuring portions 
of the desired image 136. The defect 138 in turn is 
comprised of a region 142 that only partially ob- 
scures the image 136, typically caused by the 
blurring of the defect lying on the surface of the 
film out of the image plane, or by a soft edge of 
the defect itself, and may contain a core region 144 
that totally obscures the image 136. 

Such defects 138 are typically sought to be 
eliminated in prior art by the process first of detec- 
tion followed by interpolating in the area detected 
as being defective based upon adjacent image 
information in order to mask the defect. The results 
of two prior art methods of attempting to effect 
corrections are shown in Fig. 8. In the first method 
in the left portion of Fig. 8, a human observer 
would point to the defect 138 with an airbrush and 
identify a defective area for example in the shape 
of a disk 146 of Fig. 8. The obvious problem with 
this approach in identifying the area 146 of the 
defect 138 in such a manner for purposes of sub- 
sequent manual or automated fill-in correction, is 
that it masks not only the defect 138 but portions 
of the desired image 136 as well, thus subsequent 
interpolation would remove most of the desired 
image 136 along with the defect. 

In yet another prior art method, the actual 
specific boundary and area partially obscured by 
the defect 138 is found either automatically, e.g. 
algorithmically, or by painstaking outlining of the 
defect on an enlarged projection of the image, or in 
the common art, by painstakingly applying a brush 
to the image under magnification. However, such 
an approach also has a serious drawback, referring 
to the right portion of Fig. 8, that although the area 
148 is less than that of the disk 146 which will be 
corrected algorithmically or manually (and thus less 
of the desired image 136 is lost), nevertheless a 
substantial portion of the desired image 136 is still 
lost in masking the defect, so the reconstruction of 
the desired image 136 will be imperfect. 

Finally, with reference to the present invention, 
it will be recalled that one significant benefit of 
mapping the film with infrared is so as to provide 
an infrared record which is a map of the precise 
location and boundary of the defects. Accordingly, 
in Fig. 9, with the infrared map, the outer portion 
142 (Fig. 7) of the defect 138 which only partially 
obscured the image may be restored by the in- 
frared division taught in this patent, leaving only 
that portion 150 which totally occludes the image. 



Thus the partially obscured regions 142 may be 
fully restored with accuracy, leaving only the totally 
obscured core region 150 that requires the afore- 
mentioned algorithmic fill-in. Because this latter 

s region 150 is small, little image detail is lost. Thus, 
the image interpolation across the smaller core can 
be more accurate because the image is known 
close to the area that must be interpolated. 

Additional aspects of the invention may be 

w noted. First, it has been found that even fresh film 
may contain surface waves and included bubbles 
which, although not giving rise to immediately ap- 
parent imperfections and artifacts in captured im- 
ages, nevertheless contribute to apparent graini- 

75 ness in "unblemished" film. By application of the 
invention as herein described, it has been found 
that reduction in apparent graininess to even such 
film may be achieved. It has further been found 
that benefits of the invention may be achieved 

20 essentially with all dye-based films including 
chromogenic black and white films. Moreover, al- 
though in the embodiment depicted herein, the 
sequential images captured have been digitized in 
part for convenience in subsequent processing in 

25 the manner of the invention, the invention is not 
intended to be so limited and contemplates ap- 
plication also in the analog domain. 

As an example of a specific application in 
which defects occur within the medium rather than 

30 the surface, consider the scanning of prints using 
transmitted light. When viewed by transmitted light, 
shadow detail is clearly visible that is too dark to 
be detected through surface noise by reflected 
light. In the prior art such transmission scanning 

35 was rendered useless by the random grain of the 
fiber base, however with the current invention, the 
paper fibers in the base may be effectively re- 
moved from the image along with other non-dye 
defects/providing a better way to capture a wider 

40 brightness range from prints than is possible in the 
current art. Accordingly, for the purposes of this 
invention, we consider "film" to be any transmis- 
sive substrate on which an image is layed. 

It is also not a limit to use transmitted light. 

45 When viewed under reflected infrared light, prints 
appear white with dust, scratches, and other sur- 
face defects clearly visible. By clearly distinguish- 
ing defects from image detail, and mapping these 
known defects to the visible image, the visual effect 

so of the defects may be removed. 

Moreover, a benefit of the invention is achieved 
by ensuring that the red, green, blue and infrared 
records are all captured by employing substantially 
the same optical path, e.g. from the incident light 
55 originating from the light source 42 through each 
selected filter 40, thence through the frame 10 of 
the film 12 to the camera 42 or other sensor. In this 
manner, one may be assured that each of the red, 
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green, blue and infrared images will be in registry, 
thereby facilitating ease of correlating pixels in 
each of these records to light emanating from the 
same location on the frame 10 whereupon they 
may be more efficiently processed. 5 

In the embodiment depicted herein, light of the 
various spectra has been transmitted through the 
film 12 and is collected on the opposing side 
thereof. However, the invention is not intended to 
be so limited to collection of light energy transmit- io 
ted through the medium for which correction of 
imperfections is desired. Thus the invention admits 
to applications wherein, for example, the sensor 
such as camera 42 is on the same side of the 
medium as the source of light intended to- illu- 75 
minate the medium 12, whereby the red, green, 
blue and infrared images may be captured from 
reflection from the surface of the medium 12. In 
such an instance, the sensor or camera 42 may not 
be in coaxial alignment with the medium 12 and 20 
light source 42 as in the case of the system of Fig. 
2 but rather may be off to one side of the light 
source. Nevertheless, it would still be desirable to 
illuminate the medium 12 with various light fre- 
quencies such as by laser illumination, gas dis- 25 
charge. LED (light emitting diode) or with the pre- 
viously described color filters so that the red, green 
and blue and infrared images would then be gath- 
ered by light travelling along essentially the same 
optical path for each group of such images for the 30 
noted reason of facilitating registry of image and 
defect refraction and subsequent processing of the 
red, green, blue and infrared pixels of the cor- 
responding red, green, blue and infrared images 
associated with a particular location on the medium 35 
12. 

The preferred embodiment has described se- 
quential capture of red, green, blue and infrared. It 
should be obvious that one could obtain the four 
images simultaneously using white light containing ao 
all colors by extending standard 3-color camera 
technologies to 4-colors. An example of this would 
be to use a 4 line CCD sensor with filters built over 
the lines to pass separately red, green, blue, and 
infrared, or a 4-chip camera using dichromic mir- 45 
rors to separate red, green, blue, and infrared im- 
ages separately on the 4 chips. 

Any of the elements used to recover the image 
without defects may be implemented in analog 
circuitry or by using an analog-to-digital converter 50 
with digital hardware, or even with software running 
on a computer or a digital signal processor (DSP). 
Also the several elements, such as capturing the 
red, green, blue, and infrared images, and process- 
ing the results, can be done serially or in parallel. 55 
Although the discussion has assumed serial cap- 
ture of images to a computer memory for process- 
ing, in certain fields other structures are preferred. 



For example, in film-to-video transcription, a color 
camera with a fourth infrared sensor outputs the 
red, green, blue, and infrared signals to analog 
circuitry that substantially divides the red, green, 
and blue by the infrared, providing in real time 
removal of scratches, dust, fungus growth, finger- 
prints, transparent-type splices, and other imper- 
fections. With the advent of HDTV, removal of all 
dust and hairline scratches will be much more 
critical than it is today. 

Claims 

1. Method for use in correcting for the effects of 
storage medium imperfections on an image 
stored therein comprising 

directing energy from a light source to 
said medium in the visible and infrared range, 

generating a corresponding infrared image 
and a plurality of visible images by detecting 
energy from the medium, and forming a sec- 
ond composite image as a combination of the 
infrared and plurality of visible images, thereby 
mapping the imperfection in the medium. 

2. The method of Claim 1 wherein said step of 
directing energy from said light source, in- 
cludes sequentially in any order, directing red, 
green, and blue light at said medium and then 

detecting a corresponding red, green, and 
blue image from said medium. 

3. The method of Claim 1 or 2 wherein at least a 
portion of detected energy from the infrared 
and visible range of said electromagnetic spec- 
trum were transmitted through said medium. 

4. The method of Claim 1 , 2 or 3 wherein energy 
in the infrared and visible range of the elec- 
tromagnetic spectrum is directed along sub- 
stantially the same optical path. 

5. The method of Claim 1, 2, 3 or 4 wherein said 
imperfections are comprised of a first portion 
substantially deflecting infrared energy trans- 
mitted therethrough so as to be substantially 
excluded from said detected energy; and 

a second portion only partially deflecting 
infrared energy transmitted therethrough 
whereby only a portion of said infrared energy 
transmitted through said second portion is in 
said detected energy; and wherein said second 
image is formed by estimating occluded por- 
tions of said image based on areas of said 
image partly occluded and recovered. 

6. Apparatus for use in correcting for the effects 
of recording medium imperfections on an im- 
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age stored therein, comprising 

means for directing energy from a light 
source to said medium in the visible and in- 
frared range, 

means for generating an infrared image of 
said medium containing indications of imper- 
fections with respect to said medium and a 
plurality of visible images, and 

means for generating a second composite 
image as a combination of the infrared and 
plurality of visible images, thereby mapping 
the imperfections in the medium. 

7. The apparatus of Claim 6 wherein said me- 
dium is film. 

8. The apparatus of Claim 7 wherein said film 
contains image information stored as a dye 
image. 

9. The apparatus of Claim 8 wherein said me- 
dium is substantially reflective. 

10. The apparatus of Claim 9 wherein said me- 
dium is a print. 

11. The apparatus of Claim 6. 7, 8. 9 or 10 
wherein said means for directing energy to 
said medium comprises, means for directing 
red, green, and blue light at said medium. 

12. The apparatus of Claim 11 wherein said means 
for directing said infrared energy and said en- 
ergy from said visible portions of the elec- 
tromagnetic spectrum transmit said energy 
along substantially an identical optical path. 

13. The apparatus of Claim 12 wherein said means 
for directing said infrared energy and said en- 
ergy from said visible portion of said elec- 
tromagnetic spectrum include a numerical ap- 
erture that varies with wavelength. 

14. The apparatus of Claim 11 wherein said means 
for detecting of energy from said infrared and 
said visible portions of the electromagnetic 
spectrum detect said respective energy along 
substantially coaxial optical paths. 

15. The apparatus of any one of claims 6 to 14 
wherein said infrared image is functionally re- 
lated to measurements of said infrared energy 
attenuated by said imperfections; and wherein 
said means for generating said second image 
comprises 

means for adjusting said plurality of non- 
infrared images in response to said measure- 
ments 



70 



75 



16. The apparatus of Claim 15 wherein said in- 
frared image and said second image are com- 
prised of corresponding first and second plu- 
ralities of pixels, and wherein said means for 
adjusting further comprises 

means for generating for each pixel of said 
first plurality a number functionally related to 
the infrared value of said pixel; and 

means for multiplying, for each said pixel 
of said second plurality, the magnitude of said 
pixel of said second plurality by a correspond- 
ing said number corresponding to a pixel of 
said first plurality. ' 

17. The apparatus of Claim 16 wherein said func- 
tional relationship is substantially a reciprocal. 
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